Skip to content

Add experimental uv resolver for Python lockfile generation#23212

Open
Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Affanmir:feature/uv-lockfile-resolver
Open

Add experimental uv resolver for Python lockfile generation#23212
Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Affanmir:feature/uv-lockfile-resolver

Conversation

@Affanmir
Copy link
Copy Markdown

@Affanmir Affanmir commented Apr 1, 2026

Summary

  • Adds [python].lockfile_resolver = "uv" option (default: pex) for faster lockfile generation
  • Uses uv pip compile to pre-resolve pinned requirements, then pex lock create --no-transitive to materialize the PEX lockfile
  • Supports all lock styles including universal (via uv pip compile --universal)
  • Adds [uv].args_for_lockfile_resolve for passthrough flags (e.g., --index-strategy unsafe-first-match)

Motivation

Issue: #20679 — faster, more ergonomic lockfile generation by leveraging uv's PubGrub resolver.

This builds on the uv PEX builder from #23197 (merged) and is inspired by the approach in #22949, with key improvements:

Aspect PR #22949 This PR
Universal lock style Blocked (raises error) Supported via --universal
Conflicts with main Creates uv.py (already exists) Works with existing uv.py from #23197
Unrelated changes Includes Rust file edits Clean diff, Python only
Args passthrough [uv].args (general) [uv].args_for_lockfile_resolve (specific)
register.py Modifies No change needed

Benchmarks

Environment: Apple M4, macOS 15.x (Darwin 24.6.0), arm64
Dataset: 87 top-level requirements (Trio test deps + web/data/cloud ecosystem) → 169 resolved packages
Tool: hyperfine v1.20.0
Methodology: Same dataset structure as uv's own benchmarks (Trio project requirements)

Resolver-only comparison (no Pants overhead)

Isolates the dependency resolution step — uv pip compile vs pex lock create:

Scenario pex lock create (pip) uv pip compile Speedup
Cold cache 155.9s ± 10.8s 2.7s ± 0.1s 57.6x
Warm cache 50.0s ± 8.3s 0.051s ± 0.004s 975x

End-to-end Pants comparison

Full pants generate-lockfiles --resolve=... (includes engine init, interpreter discovery, PEX CLI bootstrap):

Scenario lockfile_resolver = "pex" lockfile_resolver = "uv" Speedup
Semi-warm (wheel caches populated) 35.9s ± 13.7s 24.3s ± 0.8s 1.48x

Note: End-to-end improvement is smaller because Pants engine overhead (~20s) is constant. The uv resolver also shows much lower variance (±0.8s vs ±13.7s), providing more consistent lockfile generation times.

Correctness

Both resolvers produce equivalent lockfiles — 168/169 packages identical, with minor differences in optional/conditional dependency selection (greenlet vs tomli), which is expected behavior between different resolvers.

Reproduction

# Direct resolver comparison (cold)
hyperfine --runs 3 \
  --prepare "rm -rf /tmp/pex-bench-cache /tmp/uv-bench-cache" \
  -n "pex lock create" \
    "PEX_ROOT=/tmp/pex-bench-cache pex3 lock create --style=universal --target-system linux --target-system mac --interpreter-constraint 'CPython==3.11.*' --output=/tmp/out.json -r requirements.txt" \
  -n "uv pip compile" \
    "UV_CACHE_DIR=/tmp/uv-bench-cache uv pip compile requirements.txt --output-file /tmp/out.txt --python-version 3.11 --no-header --no-annotate"

# End-to-end Pants comparison (from pants source checkout with this PR)
hyperfine --runs 3 \
  --prepare "rm -rf ~/.cache/pants/lmdb_store && rm -f path/to/lockfile" \
  -n "pex" "./pants --no-pantsd --python-lockfile-resolver=pex generate-lockfiles --resolve=..." \
  -n "uv"  "./pants --no-pantsd --python-lockfile-resolver=uv generate-lockfiles --resolve=..."

Disclaimers

  • Results are environment-specific; performance varies by hardware, OS, and network
  • "Cold" clears process/resolver caches; "semi-warm" retains downloaded wheel caches to isolate resolver speed from network I/O
  • --no-pantsd used for all measurements (no daemon memoization)
  • The end-to-end Pants test runs from source (includes Rust engine compilation overhead not present in released binaries)

How it works

requirements (from targets)
        │
        │  [python].lockfile_resolver = "uv"
        ▼
  uv pip compile [--universal] [--python-version X.Y]
        │
        ▼
  pinned requirements.txt
        │
        │  pex lock create --no-transitive --style={universal,strict,sources}
        ▼
  PEX lockfile (same format as today)

Usage

[python]
enable_resolves = true
lockfile_resolver = "uv"  # opt-in

[uv]
args_for_lockfile_resolve = ["--index-strategy", "unsafe-first-match"]

Current limitations

Limitation Reason
No complete_platforms Not yet wired through uv step
No per-resolve overrides/sources/excludes Avoids silently changing semantics
Non-universal lock styles require single Python major.minor uv resolves for exactly one target interpreter

Code changes

File Change
subsystems/setup.py LockfileResolver enum + lockfile_resolver option
subsystems/uv.py args_for_lockfile_resolve passthrough + updated DownloadedUv
goals/lockfile.py uv pre-resolve pipeline in generate_lockfile()
goals/lockfile_test.py 6 new tests (validation + integration)
lockfiles.mdx Documentation section
2.32.x.md Release note

Test plan

  • test_strip_named_repo — unit test for index URL stripping helper
  • test_uv_resolver_rejects_complete_platforms — validates error on unsupported complete_platforms
  • test_uv_resolver_strict_requires_single_python — validates error on multi-version strict
  • test_uv_resolver_rejects_overrides — validates error on unsupported overrides
  • test_uv_resolver_strict_generates_valid_lockfile — E2E: generates valid PEX lockfile with uv + strict
  • test_uv_resolver_universal_generates_valid_lockfile — E2E: generates valid PEX lockfile with uv + universal
  • All existing lockfile tests pass unchanged

AI disclosure

This PR was created with extensive use of Claude Code (Anthropic's CLI agent). Claude explored the Pants codebase, designed the implementation approach, wrote the code/tests/docs, and ran the benchmarks. All changes were reviewed and test runs verified by the author.

🤖 Generated with Claude Code

Affanmir and others added 2 commits April 1, 2026 20:47
Add `[python].lockfile_resolver = "uv"` option that uses `uv pip compile`
to pre-resolve pinned requirements before running `pex lock create --no-transitive`.
This provides 3-4x faster lockfile generation on cold cache by leveraging
uv's PubGrub resolver. All lock styles are supported including `universal`
(via `uv pip compile --universal`).

Closes pantsbuild#20679

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@benjyw
Copy link
Copy Markdown
Contributor

benjyw commented Apr 2, 2026

Hi @Affanmir, thanks for this contribution.

I have some questions about the background, motivation, and creation of this change. It is unusual to receive a PR that directly competes with an existing, pending PR by another contributor that is already going through code review. We like to keep things collegial and friendly here, and so in this case it might have been appropriate to first discuss your change with the author of that PR, and with the maintainers of the project.

It would be helpful if you could jump on Slack, introduce yourself there, and give us a bit of background about your use of Pants, your motivation for this change, and how you created it using Claude.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants